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ABSTBACT 

The Design of Training Systems (DOTS) project was 
initiated by the Department of Defense to develop tools for the 
effective management of military training organizations. Phase 2 of 
the project, from which this report emanates, involved the design and 
development of three computer-based mathematical models and their 
validation. Volume 1 presents an overview of the activities that 
comprised the design and development effort for the three DOTS 
models, a description of the validation process, and the long-range 
implications of the development of an operational system of DOTS 
models* Includeci in the volume are descriptions of the three models 
developed, their hardware and software requirements, and some of the 
organizational implications flowing from their implementation. 
(DGC) 
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ABSTRACT 



This report consists of three volumes: Volume I presents an 
overview of the activities that comprised the design and 
development effort for the three Design of Training Systems 
computer-based models, a description of the validation process » 
and the long-range implications of the development of an opera- 
tional system of DOTS models. 

Volume II presents a detailed description of the System Capa- 
bilities/Requirements and Resources model, the Educational 
Technology Evaluation model, and the Training Process Flow 
model. Model logic design, inpul^/output parameters, and data 
base communications are discussed at a level which allows an 
analytical evaluation of each model's design. In addition. 
Level I validation scenarios are presented in sufficient 
detail to allow their duplication if desired. 

Volume III contains the model and data base program descrip- 
tions and operating procedures. Flow charts and program 
listings for the models. Interface programs, and the data 
base applications programs are presented in appropriate 
sections. 

The results of Phar>e II indicate that the selected modeling 
applications are feasible. The models' validation demon- 
strated response to realistic system variable parameters. 
It was concluded that the system of DOTS models Is Imple- 
men table and will indeed represent a significant training 
cost savings. 

The DOTS Phase II design and development tasks were performed 
by IBM Corporation for the Training Analysis and Evaluation 
Group, Orlando, Florida (Contract No. N61339-73-C-0097) . 
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FOREWORD 



This report presents Phase II of a three-phase project called "Design 
of Training Systems," undertaken In consonance with the requirements of 
Advanced Development Objective 43-03X, "Education and Training." One of 
the major objectives of the project Is to develop tools for the effective 
management of training organizations. The tools include computer-based 
mathematical models of subsystems of the training system. Phase I 
developed a functional descriptive model of the naval training system 
and Idealized concepts oriented toward a 1980 time-frame. Phase 11 Involved 
the design and development of three computer-based mathematical models and 
their validation. Phase III will Involve the verfication of the models 
and the development of their potential applications. 

Sincere thanks is expressed for the close cooperation of all elements 
of the Naval Education and Training Conmand. The response to requests for 
information was enthusiastic and in all cases helpful and to the point. 
The personnel of the data processing organization (DPSCLANT), the office 
of the Director of Training, and the individual school staffs at the. Fleet 
Training Center, Nor'olk, Virginia, were especially cooperative in their 
support of this task, 

The System Capabilities/Requirements and Resources (SCRR) model, 
the Educational Technology Evaluation (ETE) model, and the Training 
Process Flow (TPF) model, were developed by Mr. R. Yanko, Mr. H. 
Bellamy, and Mr. K. Branch respectively. The statistical analysis 
for the TPF model was designed and carried out by Mr. L. Duffy. The 
long-range implications were developed by Mr. R. Hallman. Mr. C. 
Edison developed management applications and coordinated documentation 
for Phase II. Mrs. S. Goodell and Mrs. L. Girard provided editorial 
and secretarial services. 

The Training Analysis and Evaluation Group project team members, Mr. 
H. Okraski, Dr. W. Rankin, Mr. T. McNaney, and Mr. W. Lindahl complemented 
the contracted effort by establishing organizational interfaces and by 
providing guidance. 
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SECTION I 



INTRODUCTION 

DESIGN OF TRAINING SYSTEMS PROJECT DhSCRIPTION 

The Department of Defense (DOD) is faced with maintaining a strong national 
defense posture despite declining allocations and the impact of world inflation. 
The increasing cost of complex weapons systems and supporting manpower is siQ- 
nificantly increasing the challenge of meetir.g national defense objectives. 

Approximately 14%1 of the DOD annual budget is allocated to some form of educa- 
tion or training. This represents about 25?^ of OOD's total manpower budget.. 
Obviously, a major strategic thrust of the military services is to reduce educa- 
tion and training costs to a minimum, while maintaining the required level of 
effectiveness. 

As one of its actions in support of DOD's strategic thrust, the Navy is reducing 
costs through application of advanced management techniques and tools at all 
levels of planning and control within Its education and training system. One 
of the major objectives supporting this activity Is included in the Design of 
Training Systems (DOTS) project. 

A major DOTS thrust is directed toward providing training officials, under 
command of the Chief of Naval Education and Training (CNET), with an expanded 
decision-making capability for rapid and effective response to external factors 
such as the all -volunteer force and prospects of decreased military spendinq, 
and internal factors such as the consolidation of training activities and the 
application of new techniques and approaches to training. By application of 
advanced management systems techniques to the Navy training program, significant 
Increases In both effectiveness and efficiency are anticipated. 

The DOTS project is tasked with experimental validation of the application of 
advanced management systems techniques to the CNET organization, with the intent 
of providing training officials with expanded decision-making capability. The 
project stresses a step-wise progression from systems analysis, through develop- 
ment of computer-based models of selected subelements within CNET, to formal 
recommendations for making the fully verified models operational. The objective 
of the DOTS pmject is to employ the techniques of system analysis, educational 
technology, behavioral science, and operations research to provide a set of 
tools for gathering data on the quantitative performance of the training system 
as is, and for projecting the consequences of changes to the system. 

Phase I of the DOTS project has been completed. It included: (1) a compre- 
hensive study of the Naval Education and Training System, with special emphasis 
on those functions and organizations falling under the CNET; (2) development of 
a set of strategic assumptions describing the environmental elements expected 
to be affecting the Navy in the 1980' s; (3) development of recommendations lead- 
ing to an idealized training system in terms of the projected needs of the 
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1980' s; and (4) creation of a list of candidate computer-based mathematical 
models to enhance the decision-making processes of CNET training officials. 
If further information is desired, consult the DOTS Phase I Final Report^, 

Phase II of the DOTS effort has also been completed and the purpose of this 
document is to present the results of that phase. Phase II primarily involved 
the design and development of three computer-based models selected from the 
Phase I list, and an appropriate data base for model execution and testing. 
Other significant tasks were Included; however, they will be explained In later 
subsections of this report. 

The thrust of Phase III will be towards the experimental validation of the 
models In an operational test site and the accomplishment of those tasks re- 
quired to support their operational use. 

PHASE II OBJECTIVES 

The objectives of the second phase of the DOTS project were as follows: 

To develop computer-based mathematical models of a subset of the functions per- 
formed by CNET, and to select an appropriate sits for both evaluation and 
future testing of the models. 

To Identify or, where necessary, establish a data base for use In testinq the 
models. ^ 

To Identify those areas within the Naval Education and Training System which 
could not be modeled either because the existing system was not well enough 
defined, suitable modeling techniques did not exist, or where data for model 
input or testing could not be obtained. Areas of education and training requir- 
ing additional research were to be identified. 

To derive estimates of hardware, software, and personnel required to produce an 
operational system. ^ k o 

To validate the models. 

To design additional validation/verification studies to be carried out durina 
Phase III. y 

The principal element of Phase II was the actual design and development of the 
models selected as being the most promising from the standpoint of scope, 
feasibility, and Impact on training management decision-making and planning 
processes. The fact that these models were selected from a list of potential 
models which address all of the functional areas within the training system 
for which modeling Is an appropriate management tool, facilitated the Identi- 
fication of areas needing additional stu<|y and the projection of the long-range 
implications of an operational system. 



Design of T raining Systems, Phase I Final Report. TAEG Report No. 12-1, 
December 1973. ■ < < < 
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I'llASI, II TASKS 

Amilysis of the Phase II objectives resulted in the establishment of six 
'.pocific tasks which m discussed below. 

TASK 1 - MODEL SELECTION. The model selection process of Phase II was con- 
curr-fMit with the last portion of Phase Ij therefore, the selection process 
was fully reported in the Phase I Final Report. The Phase II effort involved 
visits to the principal subcommands under CNET for identification of opera- 
tional functions and subfunctions which could be described with a degree of 
completeness sufficient to permit modeling. At the same time, potential model 
users were surveyed to obtain direct inputs as to the most pressing, immediate, 
and long-term problems experienced by each command. 

TASK 2 - MODEL DESIGN AND DEVELOPMENT. The design of each model started with 
the formulation of functional specifications; i.e., analysis of the purpose of 
the model and determination of the model outputs. The model design and the 
programming language were selected to effectively fulfill the functioial 
specifications. A detailed discussion of each model is contained in Volume r 
of this report. 

TASK 3 - DATA BASE REQUIREMENTS. The development of a data base for the models 
required that data sources be located and converted to a format acceptable to 
the data base desig.i. Most of the data were in flat-paper form, therefore, the 
reformatting was primarily a manual task. Extraction programs were written, 
however, and used with BUPERS and DPSCLANT magnetic tapes for selecting and 
merging student data, class data, no-show rates, etc., for the purpose of sta- 
tistical analysis and loading data into the data base. In addition to the 
development of appropriate extraction programs, it was necessary to develop 
file support programs which load and update the data base, and provide data 
access for the models. These programs are discussed in detail in later sections 
of the report. 

TASK 4 - MODEL VALIDATION. The model validation task was divided in the origi- 
nal project design into two levels: Level 1 validation which took place during 
Phase II ; and Level 2 Validation which was designated an objective of Phase III. 
The Level 1 validation task of Phase II consisted of: 

a. A review of the model's intent and purposes by potential users at Fleet 
Training Center, Norfolk, Virginia, in which the assumptions, general 
design, etc., were examined for reasonableness. 

b. Sensitivity testing during which the inputs were varied over a reason- 
able range to ascertain the degree of change they caused in the outputs. 

c. Variability testing to ensure that stochastic processes, when employed, 
did not cause a high output variance^. 

d. Test scenarios which tested and demonstrated the performance of the 
models during hypothetical management problem situations. Test scenarios 
are presented in appropriate sections of Volume II of this report. 

^De sign and Use of Computer Simulation Models by James R. Emshoff and Roger L. 
Sisson (The Macmi 11 an Company, New York, N.Y.) p. 204. 
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In tuldition to actual validation of the models, Tusk 4 roquired the suluction 
af i\ f.ost site locntiun. Test site selection was important for two re.3sonf,. 
( irst, the nOTS project is intended to be a starting point for the dovolopinonl, 
ul <i niinpieto net of models for training management and planning. Second, Lho 
mf)d(!lr> duve loped during tha DOTS project must be applied to other traininq 
sites if the models aro to be of gene.^al, long-term value. Therefore, the situ 
soiGcted had to meet the following criteria: 

a. Have an extensive mix of training activities to demonstrate the range 
of application of the models. 

b. Be subjected to unpredictable requirements for new courses, quotas, 
etc., to allow comparison of response times for existing systems and 
the DOTS models. 

c. Include all functions in the training development cycle from course 
design to Implementation. 

d. Be in the process of implementing or planning to Implement new in- 
structional techniques such as Individualized Learning Systems (ILS). 
The primary purpose of the DOTS model set is to assist in assessing 
the impact of proposed changes in both Instructional techniques and 
training requirements. Therefore, both types of changes ware desirable. 

e. Have an active interest In the DOTS project at the comnand level. 

f. Be general enough in level and type to be representative of a signi- 
ficant fraction of Navy training. 

g. Have access to a computer system either directly or via remote terminals. 

The Fleet Training Center at Norfolk, Virginia, has been selected as the test 
site. 

TASK 5 - DETERMINATION OF LONG-RANGE IMPLICATIONS. 

Analysis of Rejected Candidates . During model selection, potentially useful 
candidates were rejected because the system was not sufficiently defined for 
modeling, or the data required to establish a relationship between key vari- 
ables were not available, or the effectiveness of a strategy in a military 
environment was unknown. Analysis of those rejected candidates resulted in 
recommendations for further studies which should serve to identify where addi- 
tional data can be obtained with the ultimate goal of improving system control. 

Est imation of Hardware/Software and Personnel Requirements . This subtask was 
co"ncerned with projecting the cost of an operational system. Program size and 
the need for additional support programs were determined as the model pro- 
ceeded through the development stage. The assumptions about hardware and opera- 
ting environment for the system are presented in Section IV of this volume. 

TASK 6 - MODEL TUTORIAL SOUND/SLIDE PRESENTATION. The purpose of this task was 
to create a sound/slide presentation that would develop a basic understanding 
of the operations research tenn "model" within all levels of CNET management, as 
well as all operations personnel who would be involved in the implementation 
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and use of tho models developed during the DOTS project.. The presentation 
discusses the concept of mathematical modeling on a general level and does 
not midress any of the three Phase II models specifically.^ 

I'llASL II PRODUCTS 

The tasks accomplished during Phase II provided the following products: 

a. The Phase II Final Report 

I). The Training Process Flow Model 

c. The System Capabilities/Requirements and Resources Model 

d. The Educational Technology Evaluation Model 

e. The Model Data Base 

f. The Sound/Slide Model Tutorial. 

This Phase II Final Report contains the design specifications, program listings, 
flowcharts, and brief operating instructions for the three models and the data 
bases. In addition, this report contains verification plans, hardware and soft- 
ware estimates, and recommendations for additional studies of subfunctions with- 
in the training organization. Items a. and f. are the physical deliverables of 
Phase II; the work product of all Phase II tasks are contained in them. 



*The Sound/Slide Presentation is distributed by the Training Analysis and Evalua 
tion Group, Orlando, Florida 32813. 
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^CSHWN^^ SECTION II 

DESIGN OF TRAINING SYSTEMS MODELS 
OVLKVIEW OF THE DOTS MODELS AND SYSTEM DIAGRAM 

During Phase II of the DOTS project, three computer-based models (the System 
Capabilities/Requirements and Resources model, the Training Process Flow model., 
and the Educational Technology Evaluation model) were designed, developed, and 
>/a]1ddted. In addition, a data base was designed and developed as a separate 
entity which functions, however, as an integral part of the three model system. 
The models and the data base are presented in general terms in this section, 
which is intended to provide a description of their characteristics and capa- 
bilities as management tools. A detailed description of each model is presented 
in Volume II, and a detailed description of the data base is presented in 
Volume III of this report. Figure II-l presents the Phase II DOTS system. 

BACKGROUND - MODELING TECHNIOUES. Computer-based mathematical models are used 
to simulate real or hypothetical systems, or to optimize the use of a system 
for some specified objeclive. The design of a simulation model is such that 
system performance under specific conditions is duplicated. The input variables 
and constraints are manipulated by the model user, and the model provides one 
*'.olution for that particular set of inputs utilized over a predetermined interval 
of time. By an iterative process of performing manual feedback and making multiple 
model runs, the user can determine the best set of input conditions that solve the 
problem. 

The design of an optimization model is such that the internal logic of the model 
manipulates the variables to optimize the use of resources to solve a problem. 
Linear programming or other optimization techniques are used to search for the 
combination of resources that will maximize output or minimize cost for the set 
of inputs and constraints specified. 

THE SYSTEM CAPABILITIES/REQUIREMENTS AND RESOURCES MODEL. The SCRR model is a 
linear programming (LP) optimization model. The SCRR model formulates an LP ob- 
i'ective function and constraint equations from infonnation contained in the data 
base. The LP problem is then solved to optimize training complex student through- 
put and resource utilization. Basically, the model has two modes of operation. 
In the first mode, the resources; i.e., the classrooms, laboratories, instructors, 
and the appropriate constraints and limitations applicable to each, are specified, 
and the model determines the maximum student throughput and the optimal mix of 
course convenings which can be attained In a specified time period. In the second 
mode, the desired output profile is specified, and the model determines the mini- 
mum combination of resources required to produce it. The model solution, con- 
sisting of the linear programming solution and the sensitivity analysis, gives a 
total picture of the training complex output and the utilization of each resource. 
Factors are presented which Indicate the effectiveness of, and the limits for, 
manipulating each input variable without impacting the optimal solution. 

THE TRAINING PROCESS FLOW MODEL. The TPF model is a simulation model. It uses 
information contained in the data base to create an aggregated data matrix, upon 
which the execution module logic operates in order to calculate output quantities 
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which predict training system performance. The key elements of the TPF are the 
profiles of course characteristics and student characteristics by course. The 
profiles and the weighting factors associated with them were created by statis- 
tical analysis of historical data from BUPIiRS and the Fleet Training Center, 
Norfolk, Virginia. A substantial portion of the student performance data was 
not in an Automated Data Processing (ADP) form, and had to be gathered during 
instructor intarviews. Further discussion of the statistical processes used to 
establish the proTiles is presented in a later subsection of Volume I> Section II. 

Basically, the TPF starts with a course convening schedule obtained from the data 
base, or an optimized convening frequency obtained from the SCRR model. The 
profile characteristics of the student groups are then compared with selected 
course factors; e.g., age, type, priority, etc., enrollment data; e.g., utiliza- 
tion, backlog, etc., and the throughput of the training complex is predicted. In 
• addition to throughput, certain aspects of resource utilization are calculated 
. from the. predicted thr'^-u^hput versus maximum capacity figures. 

THE EDUCATIONAL TECHNOLOGY EVALUATION MODEL. The ETE model is a simulation of an 
Individualized Learning System (ILS). It is an entity flow model which deals 
with the movement of individual students through a school as they are influenced 
by media characteristics, the number of course modules to be completed, and the 
contention for facilities and instructors. The term "evaluation" in the ETE name 
refers to its intended use in comparing the relative effectiveness. of different 
ILS course designs. The ETE model uses the course curriculums, the student types 
by curriculum, and characteristics of the learning modules in the curriculum such 
as average completion time. Instructors and other resources required, etc., to 
calculate the performance of the school in terms of throughput, elapsed time, and 
use of resources. 

THEORETICAL BASIS FOR THE DESIGN AND DEVELOPMENT OF THE DOTS MODELS 

The DOTS project is oriented toward filling a gap between theory as it 
presently relates to the management of large, complex systems, and practice as 
implemeiited al present in the management of training. The Navy is attempting 
to incorporate the latest system techniques, specifically modeling, into the 
design and operation of present and future training systems. The DOTS models 
were selected by a rigorous screening process to ensure that they represented 
modeling applications that were new and untried within the Navy, as well as 
feasible in tenns of being amenable to known modeling techniques. All three 
models represented nev/ projects completely within the domain of CNET. 

MODELING - AH ESTABLISHED TECHNIQUE. It is correct to state that modeling is an 
established technique. Computer-based models have been tried and tested in both 
industry and the Navy. Systems analysts, operations research specialists, and 
managers have collectively had wide and varied experience in the development and 
application of computer-based models. 

Computer-based modeling, both optimization and simulation, has been an outgrowth 
of relatively new analytic techniques and computational hardware development. 
The discipline of Operations Research (OR) had its beginnings in the multi- 
disciplinary analysis of military problems during VJorld War II. Although it has 
deviated from its original multidisciplinary emphasis, progress has been made in 
the use of a "scientific approach" to analyzing large systems and the problems 
associated with managing them. 
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In parallel with the development of OR techniques, the digital computer has 
developed from rudimentary experimental designs into the familiar high speed, 
high density, large storage capacity computers of today. The mathematical model- 
ing techniques of OR and other mathematical techniques for simulating a system, 
whether it was a hardware system or a functional analysis of an organization, 
were found to be adaptable to being programmed into computers. This combination 
of mathematical and programming techniques has developed into a discipline 
generally referred to as computer-based modeling. 

The theoretical basis for the design and development of the DOTS models was 
embodied In the method used for approaching and accomplishing the task. The pro- 
ject team approached the design of the models from two aspects. One aspect was 
the technical process of designing and developing a model, the second aspect was 
the practical considerations of producing models that management would use. The 
technical process followed the generally accepted approach to an OR project.^ 
Three parts to the approach: selection and definition of the area to be modeled; 
design and development of the models; and validation of each model, were carried 
out during Phase II. Two additional parts; establishing control over the solution, 
and preparations fr;r 1mplementation» will be accomplished in Phase III. The key 
activity of the design task was the selection of an appropriate modeling technique 
for each problem area. 

Technical Considerations. The Naval Education and Training System, more par- ** 
ticularly CNET, and the subelements within It could be described in terms of 
personnel* resources, and their interrelations and operations over time. The 
selection of particular modeling techniques representing the fundamental design 
or type of model for each selected subelement depended upon a multiplicity of 
factors derived from the nature of the system Itself, from the objectives to be 
met by the model* and the limitations imposed by complexity and hardware con- 
siderations. 

The DOTS project team implicitly defined the system as the Fleet Training Center, 
Norfolk, Virainia* the test site. The system factors are: (1) the process 
involved; (2; the resources being used; (3) management processes, and (4) 
internal versus external environmental elements. 

The product at the Fleet Training Center, Norfolk, Virginia, is competent people. 
The process Is the conducting of approximately 125 courses consisting of "A" 
Schools, "C" Schools* and Fleet Training Courses - of these, approximately 25 
issue an NEC. The resources used consist of classrooms, laboratories, training 
devices and equipment, elements of ILS equipment, publications and materials, 
and instructors. The management processes Involve, among other things, meeting 
personnel requirements, meeting Fleet training demands, planning for growth and 
contraction, planning for the use of new training technologies, budgeting, etc. 
The environmental factors include those that are internal to the Navy, but 
external to CNET; for example, naval personnel rotation policies and career path 
designations, as well as Fleet demand and new equipment acquisition demands. 
Environmental factors also Include those that are external to the Navy; for 
example, the academic background of available recruits, military spending, and 
physical environmental factors that affect trainee arrivals* etc. 



Introduction to Operations Research by Churchman, Ackoff, and Arnoff (John Wiley 
& Son, Inc., London 1957} p. 13. 
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The objectives of the models, and the constraints imposed by the limitations of 
the modeling techniques and potential hardware, had to be considered ;Jmulta- 
neously because the designer had to formulate reasonable and prudent objectives, 
and this can only be done when constraints are taken into account. The objec- 
tive of the model was, of course, dependent upon the intended uso of the model. 
Perhaps the liujur dlscriininaliiig factor in describing model use is whether or 
not It Is desiriible to optimize some aspect of the system, or to sir.ulate all^ 
or part of the system to observe system response. In making this aiscrimination, 
certain prerequisites had to be considered. First, If an optimization model was 
to be chosen, then a particular objective function to be maximized or minimized 
had to be Identified. Second, if linear programming was used, then linear re- 
lationships between variables had to be demonstrated in order for the model to 
be valid. Third, 1f mathematical programming of linear or other types was to 
be used, then there were complexity limits to observe In order to keep the project 
. feasible. Fourth, 1f it was desirable to simulate the system or its elements in 
order to observe system response to varying inputs, then: (1) data about past 
system perfonunce had to be available, and the historical data would have to be 
subjected to various stat1st1c<il analysis techniques to determine the relation- 
ships and patterns of Influence and reaction that described system performancei 
or (2) Interrelationships would have to be estimated or manipulated by stochastic 
or probaballstic methods. 

With the consiueratlons about system characteristics, model objectives, and 
complexity limitations in mind, the project team selected the major design 
features of euch of the three candidate models. The Training System Capabilities/ 
Requirements and the Training Resource Allocation modeling applications both per- 
tained to resource management. It was determined that they would have the 
greatest benefit if combined into a System Capabi 11 ties/Requirements and Re- 
sources model w-'th an cverall objective of optimizing student throughput with 
the ruiources .vaiUMe. The SCRR Is a typical allocation nxjdelo in which the 
resource roqui^'emeot:- and availabilities are specified, and the task of the 
model Is to dttaraJne the combination of efforts which will yield the maximum 
return on tht ucj of resourcus. Given the instructor to student ratios, class 
size, course length, etc., it was intuitively obvious that the resource usages 
per convening were linear. Therefore, linear programming was selected to achieve 
the desired optimization. 

The Training Process Flow model was .*ieen as a tool for studying the flow of 
trainees thrt<ugh the naval training process rather than as an optimization of 
any ont aspect of the process. The TPF model was designed as a simulation of a 
training complex. Extensive statistical analysis of historical data was under- 
taken to establish the relationships between a multitude of variables which were 
thought to represent student capabilities'! course cliaracteristics , delivery 
system pe^fcrrftanc?, etc. Inuring the course of the data analysis, parameters 
that proved to have no relevance to system performance were screened out. The 
relationships established by analysis were Incorporated into the model logic 
design to obtain a chro.iological profile of the training complex output by 
student type. It was decided that any optimization desired by a model user 
would best be accomplished externally through repeated operation of the model 
with differe.it Input values specified. At the same time, the simulation capa- 
bility of the model would allow the evaluation of proposed training system designs. 



^Ibld, p. 275 
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Simulation was selected for the Educational TechnoUj^' Evaluation model pri- . 
marlly because the model was Intended as a tool for coi.iparing different I L5 
configurations, a task for which simulation is well suited. Only a small 
percentage of the courses taught in the training system had been converted to 
ILS; therefore, It was not clear which aspects of the configuration wauld 
prove most taxing on resources. The ETE model provided th? opportunity to 
observe the behavior of the system under different test conditions which 
could not be observed In reality. In contrast to the advantages of simulation 
for the ETE model, there were two reasons for rejecting an optimization model 
for this modeling area. First, due to lack of ILS experience, it was not clear 
as to which aspect of the system should be optimized, if any. Second, the 
students 1n an ILS contend for facilities, consequently, queues or waiting lines 
develop which are non-linear and, therefore, less adaptable to an optimization 
process. 

Manage rial Considerations . Computer-based modeling has been expanding in 
Industry and government as the management sciences have grown m influence. Thft 
thrust has been toward the creation of tools for reliable operation and planning 
In the management of large organizations; the thrust has been away from seat- 
of-the-pants" management, with its Inherent non-reproducibi lily and lack of long- 
term efficiency. There has been resistance to the acceptance of new techniques, 
however, and modeling has been no exception. Often, managers will agree to the 
efficacy of new modeling tools, but when the experts leave, the mode is shelved 
because proper groundwork was not undertaken to ensure that the model was needed 
and understood by potential users. Greater management acceptance can be 
cultivated If the project is executed properly. 

Hamnond^ has summarized some of the salient points of a modeling project and 
enumerated precautions to be taken by management scientists attempting to produce 
useful models. They may be suinnarized briefly as follows: 

a. An analysis of the organization's functions must be made In order to 
determine potential modeling areas. Then, the areas of greatest poten- 
tial are identified and a decision Is made whether or not to model each 
one. In considering potential modeling areas, the organizational 
climate should be evaluated for the degree of management support behind 
the project. 

b. The purpose of the model is defined and the specifications are generated 
with clearly defined goals in mind. 

c. The model is designed with user involvement. Input and output informa- 
tion should be in a format that is familiar to the user. 

d. The model is coded and debugged. 

e. The model Is validated, then verified by the user. 

f. Education is provided to the user in order to gain acceptance for the 
model . 



^ Do's and Don't's of Computer Models for Planning by John S. Hammond III, (Harvard 
Business Review, March-April 1574). 
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g. The model is implemented In the management process 



h. The overall concept of simplicity of scope and design 1s a guide through- 
oub model development. 

Although the guidelines listed pertain to a planning model development project, 
they are applicable to the DOTS model design and development process. The DOTS 
project team followed virtually the same process during Phase I and II, and will 
continue the process into Phase III. 

In order to gain acceptance for the project models, potential user orientation 
and user perspective on problem areas were used as benchmarks during the model 
selection process. The test site selection process was specifically designed to 
find an orcjanizational element that had the proper management support as well as 
the technical intricacy needed to properly validate the models. During model 
design, modeling techniques were evaluated with respect to potential usefulness 
of the model, and data flow into and out of \^he model was formatted in readily 
accessible, f«ii*ri«idr form. 

During Phase IIA, training courses for potential model users will be written and 
conducted. Tnese courses will bridge the gap between personnel who participated 
In the development and validation of the models and those whose first exposure 
will be to the finished product. Phase III will lay the groundwork for imple- 
mentation, and will further enhance management acceptance by implementing 
recommended refinements to model/user communications and Input/output processing. 

One can conclude from the preceding paragraphs that the DOTS project models have 
a theoretical basis in both the technical aspects of model design and the 
practical aspects of winning user acceptance of the operational models. In 
addition to technical and practical aspects, the model designs have been in- 
fluenced by conc*-,pts of decision theory and statistical analysis. These points 
are discussed in the following paragraphs. 

THE USE OF MODcLS iN DECISION MAKING. The process of decision making has been 
conceptualized as the activities that result when a potential decision maker 
receives an external stimulus. 8 The initial response is to consider the alter- 
natives and make a mental construct of the decision problem. If the alternatives 
are Inadequate to allow a clear basis for choice, additional alternatives will be 
sought. At this point, the decision maker may or may not seek data which will 
aid the prediction of the likelihood that the assumptions Involved in each alter- 
native will come true. When, in the decision maker's judgment, the evidence is 
sufficient, he selects one alternative course of action. Depending upon the 
strength of the likelihood or probability which can be assigned to alternatives, 
the decision can bu considered as falling under one of three categories: assumed 
certainty, risk, or uncertainty. 9 

It is obvious that making a decision under assumed certainty does not present a 
particularly stressful situation; the decision maker is quite confident in his 



^The Analysis of ManaaefTjent Deci lions by William T. Morris (Richard D. Em*n, Inc., 
Homewood, Illinois, 1964} p. 88. 

^Ibid, p. 10. 

II-7 

ERIC ^0 



TAEG REPORT NO. 12-2 



choice. It 1s under conditions of risk or uncertainty that methods of 
analyzing the decision situation and selecting the best alternative are needed. 

A relatively new discipline called Decision Analysis has been developed around 
this problem of making a decision under uncertain outcomes. The decision 
process has been described by one decision analysis researcher as consisting 
of: (1) a deterministic phase in which variables are defined and values assigned 
to possible outcomes; (2) a probabilistic phase which assigns probabilities to 



decision inputs in order to derive probabi 
formational phase in which estimates of va 
tlon; and (4) the allocation of resources. 



ities for the outcomes; (3) an in- 
jje are derived for additional informa- 



The Important point to express about the decision process 1s that a decision re- 
quires information whether it is perceived as a search for alternatives in re- 
sponse to a stimulus, or perceived as the process of assigning values and 
probabilities to outcomes. The DOTS models can generate the required informa- 
tion. A training official can derive alternatives quickly and imaginatively. 
He can assign probabilities to outcomes because the models, having been validated, 
are reliable predictors of training system element's response to changing input 
variables. Thus, from the analytic viewpoint of gathering, evaluating, and using 
data to reduce uncertainty in decision making, the DOTS models are a useful 
management tool. There is, however, another important aspect to decision making. 
It is the Interaction between managers and the models, and managers with each 
other In a problem solving situation. 

In order to study the effectiveness of computer-based support ^f management 
decision making, Morton'' conducted a field experiment at one division of a 
large multi product corporation. The experiment consisted of observing the 
management decision process prior to and after the installation of computer- 
based models and a display system. The goals of the project were to determine: 
(1) if the system could be used; (2) at what management level it could be used; 
and (3) what impact the computer-based models and display could have on the 
decision making process. 

The results of the experiment contain direct Implications about the expected 
usefulness of the DOTS models. Norton found that the computer-based management 
decision system, in the opinion of the managers using it, did indeed aid the 
decision making process. By observing problem-solving situations, he concluded 
that the system made problem identification easier, provided more alternatives, 
reduced the time required to make decisions, and improved communications be- 
tween managers. The improved conmuni cations between managers was considered the 
important factor for producing more creative and beneficial solutions to 
managerial problems. Thus, part of the theoretical basis for the DOTS models 
was derived from the relationship bet^jeen rapid information processing and 
efficient decision making. 



^ ^Decision Analysis; Toward Better Naval Management Decisions by Lt. N. Clark 
Williams, U. S. Navy (Naval War College Review}. 

^ ^ Management Decision Systems by Michael S. Scott Morton (Division of Research, 
Graduate School of Business Administration, Harvard University, Boston, 1971). 
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STATISTICAL PROCtSSES IN THE DOTS MODELS DESIGN. This subsection applies 
exclusively to the Training Process Flow model. The TPF model logic design 
process incorporated statistical analysis as a fundamental determinant of the 
mathofnaticai and logical interrelationships linking the input variables to 
thu model outputs. The SCRR model, as mentioned previously, used a standard 
linear programming technique which did not require a statistical analysis of 
input variables. Once the inherent linearity of resource usage was accepted, 
the problem was amenable to a deterministic mathematical solution. The ETE 
model used a statistical queuing model to determine waiting time at facilities 
and the length of queue or number of students waiting, but other parametric 
factors were determined by the user at the time of model execution. 

The determination of interrelationships among variables of the Training Process 
Mow model comprised a collection phase and an analysis phase. The collection 
r-»ffort was shaped by two factors: (1) it was necessary to obtain a large sample 
of records in order to confidently project sample characteristics into the role 
of population parameter predictors; and (2) some records were available in ADP 
form, making bulk collection feasible; others were in flat-paper form making 
collection by hand necessary. 

To obtain the records that were stored in ADP form, programs were written to 
extract data from DPSCLANT tapes and merge them with other DPSCLANT data or 
BUPERS data as appropriate. The process proceeded as follows: 

a. The social security numbers and pass/fall data for the majority of 
students attending school at the Fleet Training Center, Norfolk, be- 
tween 1 January and 30 June 1974, were extracted from the DPSCLANT 
student file - some 27,000 records In all. 

b. All courses were examined for size, frequency, and pass/fall history 
In order to select courses that might show causal relationships be- 
tween student and course characteristics and pass/fail behavior. Team 
training, orientation, and non-graded courses were eliminated. After 
evaluation, approximately 5,000 social security numbers of students 
who had attended 42 of the courses were sent to BUPERS to obtain 
student characteristics; e.g., rate, years of education, GCT score, 
ARI score, etc., from the Enlisted Master tape. BUPERS returned 
approximately 4,000 student records, representing all those among the 
5,000 who were still on active, enlisted duty. 

c. Student characteristics, pass/fail data, and manual inputs such as 
grades obtained from Instructor records, were merged into a data base 
and retained for statistical analysis. 

d. A second statistical data base was created from selected student data, 
the DPSCLANT no-show file, and course characteristics; e.g., course 
age, revision activity, priority, etc., for 110 courses. Course in- 
formation was supplied by school training staff members. 

The statistical analysis of the student and course data base files was accomplished 
using SPSS Version 6.'' First, the data were processed and presented in descriptive 



Statistical Package for the Social Sciences, National Opinion Research Center, 
University of Chicago. 
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formats; I.e., tabuUtecl, displayed in histograms, and summarized by key statis- 
tics in order to screen the variables for overall characteristics indicating 
potential relevance to the problem. The student file variables were then sub- 
ject(?d to a correlation analysis to determine the degree of covariation between 
"the 31 student/course characteristics for the 42 selected courses. In addition, 
correlations were run on 15 characteristics for all remaining courses taught 
at the Fleet Training Center. 

After the correlations were complete, multiple regression analyses were run on 
the variables. The regression analysis was used to determine the degree of 
dependence and hence, the predictive capabilities of variables. The results of 
the statistical analysis were evaluated for reasonableness In order to eliminate 
spurious associations. The attribution of a cause-and-effect relationship was 
a judgmental function of the analyst. The relationships emerging from the 
evaluation were incorporated, where appropriate, into the model logic and/or the 
data base for use by the TPF model during execution. 

MANAGEMENT APPLICATIONS OF THE DOTS MODELS 

The decision to develop computer-based models was greatly Influenced by the 
size and complexity of the Navy training organization. The system Is complex 
enough that It is difficult for a manager to keep all the relevant factors of a 
problem situation In order, and apply them at all appropriate areas of impact. 
Properly designed computer-based models, such as the DOTS models, have no 
difficulty tracking and manipulating a multitude of variables, constraints, and 
operational relationships. 

Effective training management requires that managers understand training goals and 
objectives, establish priorities, select performance standards, and establish a 
feedback and control system for training operations. Within that management 
framework, the DOTS models can be applied 1n the following general areas: 

a. Planning - to assess long-range demand estimates, to provide for 
Inclusion of new training technology, to allow for changes in student's 
educational background, to provide for alternative staffing policies, 
and to design new training systems. The DOTS models give managers the 
opportunity to test system reactions in areas where they have not had 
previous experience! without running the risk of disrupting the train- 
ing system. Thus, the probability of high quality planning is in- 
creased and the cost of correcting mistakes is reduced proportionately. 

b. Evaluation to assess the present use of resources and measure the 
effectiveness or efficiency of resource usage, to measure training 
complex sensitivity to various resource changes, to evaluate the cost 
of different training technologies, and to evaluate school performance 
in terms of failure rates, etc. 

<« 

c. Problem solving - to anticipate problems by asking "what if" type 
questions and exercising the models to get answers, and to project 
the impact of short-range changes in demand, resources, student 
characteristics, etc. 

d. Information retrieval - to gain perspective on system functions and 
performance by observing simulated system operations. In addition. 
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use of the models gives the manager rapid access to information in 
the iJrita bd^(^ If properly updated and maintainod, the data 1)Q'.(' 
is a valuabl(.' inventory of all the training resources of the coinplfx 
and the constraints upon their use. 

'.YSTKM UPAUlLITlES/REqUiREMENTS AND RESOURCES MODEL APPLICATIONS. The SCRR 
model can be applied in the following specific types of situations: 

a. Assessment of Tong-term training demand. The SCRR model in its first 
mode of operation will optimize the number of course convenings or 
student throughput w1th1n stated resource constraints. It can be 
used, therefore, to determine whether annual training requirements are 
feasible. If demand is projected beyond the coming year, the SCRR 
model can signal the need for additional facilities before present 
facilities are exhausted. The optimized convening rate can serve as 

a guideline for course scheduling. 

b. Assessment of the impact of short-term demand that might arise from 
unscheduled events, such as a ship repair operation, an activation of 
reserves, or unusual seasonal recruitment levels. In these instances, 
the SCRR model maximized throughput by course would serve as an 
immediate indication of training complex capability. If necessary, a 
training manager can alter the present course convening schedule, 
deleting low priority courses to gain classroom space, and possibly 
Instructors, for additional sessions of high priority courses. 

c. Assessment of the use of training resources. In Its second mode of 
operation, the SCRR model will take the current throughput rates and 
determine the optimum combination of resources required to produce 
their. In this mode of operation, the model output can be compared 
with real resource utilization to obtain estimates of the efficiency 
of training complex resource use. 

d. Comparison of alternative training Implementation strategies. Either 
mode of operation may be used to evaluate different combinations of 
training technologies (when average- time- to-complete, etc., are 
supplied). In addition, the sensitivity analysis gives an indication 
of the sensitivity of the training complex throughput to each resource. 
As explained In Volume II, page il-7, sensitivity factors indicate the 
range over which the resource may be manipulated without affecting the 
optimum convening/throughput rate. The training manager can easily 
determine the limiting resource for any particular set of conditions, 
and apply his energy effectively by dealing with the most crucial 
problem. If, for example. Instructor availability proved to be the 
limiting factor on one course, cross-training of present staff might 
prove to be the most cost-effective way to Increase school throughput. 

In summary, the SCRR model has two basic modes of opera.tlon. In the first mode, 
training complex throughput is maximized within specified constraints and avail- 
able resources; In the second mode, the throughput by course is specified by the 
user, and the model outputs the optimum (minimum) mix of resources required to 
produce that throughput. By using one or the other of these modes of operation 
as appropriate, the training official or training staff member may plan for meet- 
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in<j projected demand, solve resource use problems, or assess ditfcront training 
i mp 1 omenta t ion strdteqie'"> . 

IRAINING mass FLOW MOULL APPLICATIONS. Although the TPF model is 'ntonded 
.r. a resource utilization control tool similar to the SCRR, because its design 
incorporates student characteristics and additional course information, its 
dpplications are significantly different. The TPF model can be applied in the 
following specific types of situations: 

a. Simulation of the training complex to determine the accumulated effects 
of demand. In this type of application, the TPF will assess the 
average-on-board, the training complex throughput, and the student back- 
log that builds if demand exceeds the enrollment capability. 

b. Assessment of overutilization or underutilization of resources at the 
course level. In this application, the model is used to evaluate the 
effects of increasing the demand for a particular course. Evaluation 
of the capacity, utilization, and no-show data will determine the need 
for scheduling additional sessions of the course or tightening the 
input requirements and the methods of reserving space in class. 

c. Analysis of the effects of student characteristics on performance. The 
TPF can be used to determine the effect that changes in student academic 
background have on course failufe rates and academic set-hacks. In this 
type of application, the model will simulate throughput with the GCT 
scores, ARI scores, time in rate, etc., specified. Training managers 
can determine which type of student, among those available, optimizes 
utilization of the course, and reduces failure rate and academic set- 
backs to the lowest practical levels. In this mode of operation, the 
training official can evaluate the performance versus resource consumption 
statistics for each course, and' calculate the efficiency of each school 
within the training complex. 

In sutrmary, the Training Process Flow model can be used in the analysis of resource 
utilization at the training complex level, or at the individual course level. The 
TPF can assess the effects of changing the student quantity and/or academic back- 
ground. As a simulation tool, the TPF allows the training manager to evaluate 
different training resource utilization strategies in terms of overall training 
implementation efficiency. While the SCRR can determine the maximal throughput 
based on total class capacity and convening frequency, the TPF can predict actual 
throughput based on the maximal throughput, student attrition, and no-show data. 

FDUCATIONAL TECHNOLOGY EVALUATION MODEL APPLICATION. The ETE model is applicable 
to problems that arise in the design and management of Individualized Learning 
Systems (ILS). As a simulation of the flow of students through an ILS, the ETE 
can be used in the following specific types of situations; 

a. Projecting resource requirements for an its course design. The ETE 
model can be used to simulate the operation of a new ILS course for 
the purpose of determining the number of instructors, instructional 
equipment, classroom space, cerrels, etc., that are needed to support 
the estimated student load for the course. 
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h. Pro.jt'rtinij throughput for an ILS course. The ETE will project tho 
numb'^r of tu.di;nts completing th«> course and tho avGrago-timo-Lo- 
comp for all students. In this application, the training marit^ficr 
specifies tho f'cicilities and trrnning resources that are availablo, 
and the model d-Uerimnes the student completion rate, waiting liiu.'s, 
etc. 

c. Evaluating the effect of changes in staff and equipment. The training 
manager can evaluate the effect of instructor cross-training on student 
flow and compay*fe the benefit versus cost of staff upgrading. The hTE 
will also indicate the effect that additional resources have on wait- 
ing lines, t1me-to-complete, throughput, and facilities utilization. 

d. Assessment of the cost-effectiveness of different ILS configurations. 
The training staff can use the ETE to run comparisons of student flow 
with different uquipment being used to support the learning modules 
within a curriculum. Measures of effectiveness of the configurations 
can be projected from cost per student calculations based on throughput 
versus the cost of resources consumed. 

In summary, the Educational Technology Evaluation model can be used during the 
design stage of an ILS curriculum, or during operational Implementation of an 
existing ILS school. The ETE can be used at the school level for evaluating 
resource utilization, or at the training complex level for assessing the cost- 
effectiveness and feasibility of converting from conventional lockstep instruc- 
tion to some form of individualized instruction. 

GENERAL OPERATIONAL REQUIREMENTS AND LIMITATIONS OF MODEL APPLICATION. Use of 
the education management models being developed for the DOTS project is subject 
to the same limitations and requirements as the use of any computer-based model: 

a. Proper use of the mod'il requires trained personnel who are familiar 
with the training system, the model design, and the proposed use of 
the outputs. 

b. The model does not give an absolute "right" answer. It gives an answer 
based on the variables and constraints specified by the user. The user 
must understand the degree of reasonableness of the variables specified. 

c. The model data base must be updated by parts of the system outside the 
model. The model output can only be useful when data base information 
is timely and accurate. For this reason, it is imperative to provide 
for an information flow into the data base. 

d. Models are simplifications of reality and they must be treated as such. 
Thpir use requires good judgment. They do not relieve the manager of 
the ultimate responsibility for decision making. It must be emphasized 
that the model is only a tool. The manager/user must evaluate many 
aspects of the alternatives that are not included in the model. Environ- 
mental factors, such as Intangible preferences, and guidelines must be 
taken into consideration. The judgment and intelligence of the manager 
will determine the operational effectiveness of the models and is the 
key to their eventual integration Into the management system. 
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MOUI.L OliJtCTIVLS AND GENERAL FUNCTIONAL SPECIFICATIONS 

The model objectives and general functional specifications were an outgrow t 
of tho development of a list of candidate modeling applications during DOTS PhrK, 
I. I'hi; .jandidate list was constructed by a two-fold process involving inU.Tviow 
with potential users and an analysis of the functional model of the Naval [''.Jucd' 
lion and Training Command. The objectives and the general functional specifica- 
tions for each modeling application were determined by putting each candidate 
into perspective against a framework of all potential and existing models within 
CNET. The framework formed a composite model of the total training system which 
if developed, would be capable of predicting the behavior of the system under a 
variety of conditions. 

The DOTS project was constrained, however, to develop only a small subset of all 
the possible candidate modeling applications. The objectives and general 
functional specifications for each of the three candidates selected for further 
design and development evolved during the Phase I conceptualization, and were 
clarified during the early, definitive design stages of Phase II. They are de- 
scribed in tiie following paragraphs. 

SYSTEM CAPABILITIES/REQUIREMENTS AND RESOURCES MODEL OBJECTIVES AND GENERAL 
FUNCTIONAL SPECIFICATIONS. The objectives of the SCRR model are: 

a. To assess the feasibility of meeting annual training requirement: at 
the training complex level in terms of desired school throughput. 

b. To evaluate alternative training Implementation plans and their effect 
on the training organization. 

c. To address instructor billet utilization at the school level as a 
function of training requirements. 

d. To assess the utilization of existing resources in the daily opera- 
tion of the training complex. 

from the model objectives, overall functions were specified for the SCRR model: 

t Maximize the student throughput based on the optimal mix of course 
convenings which the training complex can achieve in a specified 
period of time within the resources available. As a subfunction of 
this optimization process, the model should evaluate resource utiliza- 
tion, and analyze the sensitivity of the throughput to each resource 
involved. 

t Minimize the resources used to achieve a particular, stated output-mix 
objective. As a subfunction of this process, resource utilization 
should be evaluated also. 

r.UlJCATIONAL TECHNOLOGY EVALUATION MODEL OBJECTIVES AND GENERAL FUNCTIONAL 
SPECIFICATIONS. The objectives of the ETE model are: 

a. To provide information for educational techrulogy trade-off analyses 
based on variables which determine the costs of training associated 
with utilization of various educational strategies. 
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b. To provide information which will allow design of tho trdinifig process 
that will produce the desired annual output while minimizing required 
equipment. 

From the model objectives, overall functions were specified for the ETE model: 

• Simulate the flow of students through a self-paced, individualized 
learning environment. 

§ Given curricula and media descriptions, an estimate of the input rate 
for different student types, an inventory of instructors, learning 
modules, and facilities* project system output, average time-to- 
complete! and instructor and facility utilization. 

TRAINING PROCESS FLOW MODEL OBJECTIVES AND GENERAL FUNCTIONAL SPECIFICATIONS. 
- The objectives of the TPF model are* 

a. To provide for analysis of the relationship between school throughput 
rate and student arrival rate, course scheduling, capacity utilization, 
etc. 

b. To provide Information which will allow design of the training process 
that will produce the desired annual output wh .a maximizing utiliza- 
tion. 

From the model objectives, overall functions were specified for the TPF model: 

• Simulate the flow of students through a collection of schools. 

• Wherever possible, project course output based on the student input 
characteristics and expected performance. 

. a Produce a time-oriented profile of the training complex output mix. 
DATA BASE GENERAL DESCRIPTION 

Data bases are conmon data storage banks designed for applications where more 
than one program must have access to the same data elements. Prior to the emergence 
of the data base concept, new programs were generated Independently of any already 
in existence. The data used by each program were specified and entered into 
storage by the programmer, and such data could only be accessed by one program. 

Development and use of a data base for a program system, whether it is a manage- 
ment information system or a group of related models, results in higher effi- 
ciency of data operations. Data base operations have less redundancy of data 
storage, greater speed in updating data, and allow the updating of all data in one 
update program operation. Consequently, the overhead involved in data maintenance 
is reduced, and the models can be applied to a problem sequentially without the 
need for multiple data crosschecking. 

The data base developed for the DOTS project has two overall functions. The 
primary function Is to interfact with the models to supply the variables and 
constan:s used In the execution of mathematical and logical operations (see 
Figure II -1, System Diagram). Only two of the models developed during Phase 
II, the SCRR and the TPF models, interface with the data base; the ETE model 
does not require a data base, primarily because of a lack of historical data 
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about Individualized Learning Systems within the Navy. The secondary function 
of the data base Is to serve as a stand-alone, automated inventory of training 
resources which will provide timely resource information for the training staff. 
A detailed description of the data base structure and contents is contained in 
Volume III, Section V, of this report. 

The functional requirements for the data base system were established by the data 
needs of each model. These requirements fall into three categories: data base 
content* data base structure • and data base maintenance. 

DATA BASE STRUCTURE. The data base system consists of application programs, a 
file management/data base interface system, data files, and associated procedures. 
These elements comprise a total system which operates in a batch processing en- 
vironment to satisfy the data base functional requirements. 

The system contains the model -requi red and narrative data, in either numeric or 
character form, in two data files for ease of maintenance and design simplicity. 



The two data files are called the "course file" and the "instructor file." 
Student data is stored in the course file under appropriate course numbers. The 
complexity of the data elements composing each file Influenced the structure of 
the file itself. The Instructor file has a simple structure consisting of only 
single segments, which contain no dependent segments or additional information. 
The course file, however, has a complex, hierarchical structure based on parent 
segnents and subordinate segments for each data element in the file. 

The files' structures can be described briefly as follows. The Instructor file 
is a simple sequential file containing Instructor name, rate, department, re- 
porting date, rotation date, and availability. The course file Is a complex file 
containing: (1) a parent segment with the course number, course name, depart- 
ment, length, number of convenings, etc.; and (2) subordinate segments for re- 
lated courses, student/instructor ratios, qualified instructors, classrooms avail- 
able, student group characteristics, and the statistical regression coefficients. 
The two files, collectively, hold all the information required by the SCRR and TPF 
models. As mentioned previously, the ETE model does not access the data base at 
this time. 

DATA BASE MAINTENANCE. Five basic application programs; load, update, print, dump, 
and restore, perform the data maintenance function for each data file. As the 
name implies, the load programs load each file either initially or upon file re- 
organization. The load programs read data from punched cards, reformat the data, 
and perform limited error checking. The update programs serve a similar purpose, 
but will provide the capability to add, change, replace, or delete data from the 
data base. 

The file print programs retrieve all data from the data base and print a formatted 
file report. This report can be used to examine and update the files. The dump 
program and the restore program are used In the creation of the scratch data base 
or the reorganization of the data bases. The scratch data base supplies problem 
oriented data to the model Interface programs. 
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MODEL/DATA BASE INTERFACE 

The Interface programs between the models and the data files comprise the 
communication link between the model logic, per se, and the data base. The inter- 
face programs perform three functions. First, the data elements required by the 
model are extracted from the total data contained in the data base. Second, 
simple calculations are performed on the selected data. Third, the data and/or 
the results of the calculations are reformatted and passed to the model logic. 
In the case of the SCRR model, for example, the resource requirements algorithm 
extracts instructor group, the hours instructors are available, classroom numbers, 
etc., and builds the resource requirements matrix which is used to formulate the 
inputs for the linear programning module. 

The interface program for the TPF model performs the same general functions of 
extraction, calculation, and reformatting of three types of data. One routine 
accesses course data for scheduling purposes, a second routine accesses student 
data and calculates performance -factors, a third routine extracts nomenclature 
and other Information for the output routines. 

Data transfer and conmuni cation is a undirectional process from the data base to 
the models; model results are not automatically loaded into the data base, they 
are printed and stored in memory for further use. This design preserves the 
integrity and accuracy of the data base. If it is desirable to alter the data 
base contents, the update application program Is used. 

MODEL INTERACTION AND SEQUENTIAL APPLICATION 

The terms interaction and sequential operation are used because the models 
Interact through indirect transfer of the output information of one model to 
the input information for another model. This Implies that the model user is 
involved in the interaction of the models, and that is indeed true. An inter- 
mediate, user performed analysis and Interpretation of the results of a model 
application using the first model is required. The Interpreted output is used 
In posing questions and formulating a problem for the second model. The problem 
IS formatted in terms of the input variables of the second model, and the 
results of the second model's operation are in turn analyzed and interpreted by 
the user. At this point, it may be desirable to run another iteration with the 
same two models, or possibly, use the third model for additional information. 

The sequence of model operation is determined by the conceptualization and 
definition of the problem. The number of iterations and the precision of the 
results are a function of experience with problem formulation and model applica- 
tion. 

In general terms, the function of each model, as an element of the set of models, 
may be thought of as providing inputs, or constraints, upon input data for another 
model. For example: 

a. The SCRR model projects throughput based on the number of convenings for 
each course in the time specified. This is based on the assumption that 
all students complete the course. The TPF will constrain or refine this 
throughput value with student performance factors in order to predict 
"actual" throughput. 
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b. The TPF operations are dependent upon a convening schedule produced 
within the model. The SCRR can determine if the desired number of con- 
venings is feasible with the training resources available and thus, 
the output is realistic and useful. 

c. The ETE model interacts with the SCRR model through the exchange of data 
on resources available, results pertaining to average-tirne-to-complete. 
instructors used, etc. 

The number of combinations of sequential applications is highly dependent upon 
the acceptance and use of the DOTS models by training offic als. It is obvious 
that model interaction hinges on the complexity of the problems or plans that are 
being evaluated. The maximum benefit will be obtained when problems of a com- 
plexity that precludes solution by other means are used with the models. 
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VALIDATION PLAN ^ 

GENERAL VALIDATION APPROACH 

The general validation approach was to validate the models during three 
stages of the design and development process. First, during the initial stages 
of design and development, the modelers discussed their general design concepts 
and alternatives with potential users of the models. It was necessary to 
ensure that the model being developed was appropriate to the user environment. 
Areas that were examined included: (1) the appropriateness of the input variables 
in terms of representing the significant parameters in the user's problem solving 
situation; (2) Input data availability; (3) significant relationships to be 
simulated; and (4) definition of the objective function to be optimized. The 
essential element of this pliase of model validation was the determination of how 
the training complex actu'lly operdted. How were priorities established? Which 
operations were defined in terms that allowed mechanization In model form? 
Which operations were too subtle, too unpredictable, or so Infrequently used 
that modeling them was Inappropriate, etc.? Each modeler used these discussions 
to guide the development of his model. 

The second stage of validation consisted of the evaluation of the internal opera- 
tion of each model. The tasks involved were model dependent rather than part of 
a uniform testing procedure. The Educational Technology Evaluation (ETE) model 
is written In General Purpose System Simulator (GPSS) programming language. 
GPSS, by Its transactional nature, tends to produce a monolithic or one system 
model, as opposed to a mooel composed of a number of subsystems with data flow 
between. This Unguage characteristic, combined with the aeneral purpose entity 
flow design, resulted in a situation that required evaluation of the Internal 
operation either from the total model operation aspect, or from a detailed entity 
flow aspect. Both were accomplished. The ETE was evaluated with a trace program 
to ensure that every student who entered the school followed the correct flow 
path for his curriculum and used the proper resources for each selected learning 
module. All possible paths were examined. Next, the ETE model was evaluated on 
a macro level to ensure that model results were reasonable for each test problem. 
The ETE model, being a generalized simulation Intended to aid the design of 
future Individualized Learning Systems (ILS), presented a difficult validation 
problem. When the system being modeled does not exist at the time of validation, 
the modeler must rely on judgment for evaluation of what is and what is not 
reasonable. Therefore, the model solutions had to be reasonable In terms of all 
the Information that could be gathered about the expected operation of an ILS 
school . 

Fortunately, there was an opportunity to run the ETE model with- Input data which 
were being used in the design of a new individualized Electronics Warfare (EW) 
school. A specialized EW model 13 had been developed to aid that specific task. 
Running both models with the same Input data resulted in very similar output 
values; this comparison of two different approaches (generalized vs. specialized) 
to the same problem strengthened confidence In the validity of both models. 



l^The EW model was developed by the Training Analysis and Evaluation Group, 
Orlando, Florida. 
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The SCRR model used an established mathematical programming product. Therefore, 
the validation did not require testing of the Internal operation of the model, 
beyond checking of the data manipulation routines which prepared Input data 
obtained from the data base and formatted model outputs. This task was accom- 
plished by manually cross-checking model input data with the data base contents, 
and comparing model outputs with solutions that were hand calculated. After the 
data handling routin.?s were validated, the emphasis was placed on testing the 
entire model with te.t problems. 

The Training Process Flow (TPF) model has a well defined subsystem structure. 
Therefore, internal validation was centered at a subsystem level, and was focused 
on the functional performance of the subsystems and the screening techniques used 
to trap input data which exceeded values that were acceptable to the model (see 
Volume II, Section IV). Examples of some of the functions tested included: 

a. Releasing students for courses by properly controlling backlog, no- 
shows, BUPERS seats, and substitute quotas. 

b. Establishing an on-going situation by prerunning the quarter prior 
to problem start. 

c. Determining pass/fall, setbacks, non-academic dlsenrollments, AOB, 
and student days of training. 

li. Establishing conditions at the end of the quarter. 

ij. Establishing year-end totals and outputting a report. 

After the subsystems were tested and Integrated, the data flow between sub- 
sections was tested by running test problems using the entire model. 

The third stage of validation was to evaluate the overall performance of each 
model by running hypothetical problem situations with realistic or, when 
possible, liistorical Fleet Training Center (FLTRACEN, NORVA) data. The test 
scenarios that comprised those model tests are discussed in Sections II, III. and 
IV of Volume II of this report. 

EXPECTED PERFORMANCE CHARACTERISTICS 

The goal of model validation was to ensure that the SCRR, ETE, and TPF models 
met their respective objectives, and performed realistically and suitably In a 
problem solving situation. There were no specific numerical criteria or specifica- 
tions to use In Judging performance; consequently, each modeler had to subjectively 
determine the level of performance that Indicated the model response to Input 
changes was adequate, but not too large. This type of determination requires an 
analysis of the system being modeled, as well as an analysis of the model s output. 
Problems arise within that area concerned with system sensitivity because real 
systems, especially organizations, often do not perform at the margins of their 
capability. This means that historical data on system performance may Indicate 
only an insignificant change In system output for fairly large changes in Input. 
The slack or excess capacity within the system absorbs input change unless it is 
large enough to exhaust one or more of the system resources. The modeler has to 
determine the happy medium between modeling an Imperfect system exactlngly, or 
modeling the system as If it were perfect. In sunwary, the expected performance 
characteristics of the DOTS models were comprised of model objectives as presented 
on page 11-14, performed within a reasonable degree of real system performance. 

III-2 
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LONG-RANGE IMPLICATIONS 

RESOURCE ESTIMATES FOR AN OPERATIONAL D0T5. SYSTEM 

This section presents the results of DOTS Phase II, Task 5, Determination 
of Long-Range Implications. The objective of Task 5 was to define the antici- 
pated resources required to support projected implementation of the SCRR, ETE, 
and TPF models in an operational environment, A strategic plan for developing 
a multi-level system of models was also developed as a subtask of Task 5. 
Necessarily, both the near- term plan for model implementation and the strategic 
plan for multi-level development are pr?idicated on positive predictive valida- 
tion results during DOTS Phase III. 

Phase II emphasized the beic design, development, and experimental validation 
of the SCRR, ETE, and TPF models. Due to the substantial interaction with the 
training officials and st tff at tlie Norfolk Fleet Training Center during Phase 
II, and the use of actua. historical data for experimental validation, it is 
anticipated the major portion of the Phase III effort will be devoted to 
activities leading to operational implementation, with predictive validation 
representing a minority effort. Therefore, the hardware, software, and per- 
sonnel resource plans presented here assume full Implementation of the three 
Phase II models at the end of ?hase III. 

The strategic plan for developing a multi-level system of models was based on 
the long-range thrust of the current Technical Development Plan P43-03 (POIA) 
and the DOTS Phase I report's'* functional recommendations, strategic assump- 
tions, and candidate model survey. These considerations were merged with the 
Phase II validation results as a basis for developing the strategic plan 
presented In this section. 

SYSTEM Dt^/ELOPMENT AND IMPLEMENTATION. The resource estimate for an opera- 
tional DOTS svstem is divided into two major categories. The first addresses 
a system comprised of the three Phase II models and their associated data base. 
This category is based on near-term planning. The second category deals with 
suggested plans for long-range programs leading to a complex of multi-level 
models supporting a decision analysis based training management system. 

Based on various cost factors developed for categories 1 and 2, a fon^ard 
pricing exercise was performed that projected an estimated cost for 
implementing the three models and data base at appropriate locations. The 
cost exercise is documented under Cost Estimate - Int egrated DOTS Svstem> 
page IV-9. 

Category 1 - SCRR. ETE. and TPF Resource Requirements . The Phase II models 
were developed and validated on an IBM System/360 nodel 40 6F. Figure IV-1, 
page IV-2, presents an overview of that system, including programming support. 
Most of the hardware and software components were utilized In either the develop- 
ment of the models, or are required for their operational use. To enable 



"^•^ Pesiqn of Training Systems, Phase I Final Report , TAEG Report No. 12-1 
Deceiid)er 1973 
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SOFTWARE 

08/MFT - OPERATING SYSTEM/MULTIFROORAMMING FIXED TASKS 

BAL - BASIC ASSEMBLER LANGUAGE 

PL/1 - PROGRAMMING LANGUAGE/1 

COBOL - COMMON BUSINESS ORIENTED LANGUAGE 

FORTRAN IV - FORMULA TRANSLATION LANGUAGE 

MPSX - MATHEMATICAL PROGRAMMING SYSTEM - EXTENDED 

GPSS V - GENERAL PURPOSE SIMULATION SYSTEM 

SPSS - STATISTICAL PACKAGE FOR SOCIAL SCIENCES 

DL/1 - DATA LANGUAGE/1 

AUTOFLOW - FLOWCHARTING SYSTEM 



FIGURE IV-1 
IBM SYSTEN/360, MODEL 40 GF 
IV-2 
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discrimination between development and operational requirements, the two 
areas have been separated for purposes of estimating resources. 

Fi'jure lV-2\ page lV-4, and Tigure IV-3, page IV-b, present the system re- 
quiruniunts based on the assunption that the three models and data base will 
become operational without the supporting programs projected for development 
under DOTS Phases IIA and III. Resources are designated on the two figures 
as, "D" - required in the Phase II development, "0" - required for opera- 
tional ube, and "M" - required for operational maintenance. 

The planned Command and Control Center Task of DOTS Phase IIA is intended to 
embed the three models and data base in a time-sharing system, with access 
beintj accomplished through a teleprocessing terminal. The estimated resource 
requirements of the terninal and time-sharing service will be covered as a 
separate exercise in this section. The estimate to follow will ignore the 
Command and Control project. It will also assume that no significant systems 
resource requirement impacts will result from the impending Phase III effort. 

Based on the system's requirements outlined in Figures IV-2 and 3, an esti- 
mated monthly and annual cost for various patterns of data base and model 
operation was developed. From a systems perspective, the models can be 
exercised independently or independent of the data base. Although actual 
software and hardware current monthly rental rates were applied to the 
development of these resource costs, they should be treated as approxima- 
tions only. The foi lowing were points considered in developing the costs 
in Figure IV-4, page IV-6. 

a. Costs reflect only monthly hardware and software costs. No 
occupancy, personnel, expendables* power, etc., are included. 

b. It was assumed the entire system cost would be charged to the 
data hase and/or models as opposed to being distributed over 
a nuir^er of other user applications. 

c. The r.osts are based on a minimum system capability to exercise 
the n.odels within practical time limits. 

It is unlikely that any training activity, agency, or complex will require 
or couid justify a dedicated system for the sole purpose of exercising the 
SCRH, ETE, and TPF models. For this reason, an effort was initiated on 
30 August 1974, to investigate the application of the time-sharing concept 
to model utilization. Use of a teleprocessing terminal in a Command and 
Control Center environment would permit greater flexibility in incorpo- 
rating the current and future models into a decision analysis system. This 
project is designated DOTS, Phase IIA. 

Whereas the previous resource estimate. Figure IV-4, was based on the as- 
sumption of a totally dedicated system, with no part of the cost being 
shared by other applications, the time-sharing or "Cotimand and Control 
Center" pricing to follow assumes shared costs with other applications 
and, from that standpoint, represents a more valid projection of true cost 
to an individual using location. 
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IJNIT/FEAnmC DESCRIPTION 
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*ETE development and validation was accomplished within the available core. 
However, 262K (200K available) i s . recommended for optimum operational use. 
This was considered in estimating future requirements. 

D - Required during development 
0 - Required for operational use 

FIGURE lV-2 

HARDWARE UNIT AND FEATURE REQUIREMENTS 
PHASE n MODELS AND DATA BASE 
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UNIT/FEATURE DESCRIPTION 
IBM SYSTEM/360 MODEL 406F 
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'SPSS - National Opinion Research Center, 6030 Ellis Ave., Chicago, 111. 60637 
^Autoflcw - Applied Data Research, 2425 Wilson Blvd., Arlington, Va. 22201 

"M" - Designates programming revision and maintenance as opposed to day-to-day 
operations. 

FIGURE IV-3 
SOFTWARE - REQUIREMENTS 
PHASE II MODELS AND DATA BASE 
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SOFTWARE 


HARDWARE 


. COMBINED 




MONTHLY 


ANNUAL 


MONTHLY 


ANNUAL 


nuw 1 nu T 




DATA BASE ALONE 


654 


7,848 


16,670 


200,040 


17,324 


207,888 


SCRR ALONE 


736 


8,832 


15,337 


184,044 


16,073 


192,876 


ETE ALONE 


56 
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17,137 


205,644 


17,193 


206,316 


Tnr fli nur 

TPF ALONE 


720 


8,640 


17,177 


206,124 


17,897 


214,764 


DATA BASE AND SCRR 


786 


9,432 


16,776 


201 ,312 


17,562 


210,744 


DATA BASE, SCRR AND ETE 


842 


10,104 


18,576 


222,912 


19,418 


233,016 


DATA BASE, SCRR, ETE AND 
TPF 


908 


10,896 


18,977 


227,724 


19,885 


238,620 



NOTE: SPSS represented a one time charge of $600 as opposed to a monthly rental. 
This was pro-rated at $50.00 a month over one year. Therefore, the Data 
Base and TPF monthly software charges will drop $50.00 after the first year. 



FIGURE IV-4 
ESTIMATED HARDWARE AND SOFTWARE RENTAL COSTS 



IV.6 

39 



TAtG REPORT NO. 12-2 



The Phase IIA Command and Control Center task is scheduled for completion on 
28 February 1975. The Phase IIA final report will contain an accurate price 
projection based on actual experimentation with an Installed terminal and 
time-sharing service. However, a tentative forward pricing estimate has been 
developed and is presented here for information purposes. 

Based on a typical large training activity using the models and data base 
In Its decision process, it is estimated that terminal, line, and time- 
sharing CPU total cost will run about twenty- thousand dollars per year. 

The above estimate presupposes that most actual model runs will take place 
during the time-sharing services low load shifts. The terminal will be used 
to Initiate model r^m requests, to selectively display stored run results, to 
Input or update data, and to extract data. In those cases where the entire 
nwdel run Is required, a mail run will be requested as opposed to printing 
the entire run through the terminal printer. The time-shared system is in- 
cluded as one of the alternatives In the subsection entitled. Cost Estimate- 
Integrated DOTS System , page IV-9. 

Category 2 - Strategic Plan Multi -Level Model Development . The long-range 
thrust of Technical Development Plan P43-03 (POlA) is towards the Integration 
of computerized mathematical models into all appropriate areas of the CNET's 
decision analysis process. This thrust implies change in the w^y decisions 
arc currently being made if the total decision process Is to derive maximum 
benefit from the models. Therefore, an orderly and logical implementation 
of both the models and changes to the decision process itself are essential 
if the ultimate results are to be accepted and effectively used. 

The resources defined under Category 2 apply primarily to the programs re- 
quired to achieve this orderly and logical transition, rather than to those 
resources required to provide operational software and hardware coverage for 
the completed total DOTS modeling system. To attempt such a speculative 
exercise would produce costs of such limited accuracy as to be valueless. 
However, on a unit cost basis, it can be assumed that the operational sys- 
tems cost of more sophisticated models will not greatly exceed that of the 
SCRR. ETE, or TPF models. 

For purposes of estimating the resources required for the design, development, 
validation, and implementation of a total CNET's decision analysis system, 
models were arbitrarily divided into three levels. The hierarchy was based 
more on the functional use of the models than on the level of training com- 
mand using the results of the models, although the two do tend to equate. 
Figure IV-5, page IV-8, provides an overview of the three levels. 

The SCRR, ETE, and TPF models are examples of the first level. They are 
primarily concerned with the projection of training resources and student 
flow. The development of a multi -level DOTS modeling system was initiated 
at the first level, since th« horizontal Implementation of these models 
across the Naval Education and Training System will require standardization 
of a basic data base and associated procedures. Such a standard will be 
essential to the support of higher level models. This same advantage will 
apply to the evolution from the second to the third level. 
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PROJECTED DOTS MULTI-LEVEL MODELING SYSTEM 
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The second level model Is Intended to serve as a fnedlating link between 
first and third levels. The third level Is concerned with strategic con- 
cerns, extending over distant time horizons, having little to do with near- 
term operational CNET planning, but having an eventual Indirect Impact. The 
second level model will consider these Impacts, as well as the projected 
Impacts of anticipated training strategies and technologies, and convert 
them to a parametric form acceptable as Inputs to the first level resource 
models. 

Figure IV-6, page IV-10, provides an overview of the anticipated schedule of 
programs leading to an Integrated DOTS multi-level modeling system. Develop- 
mental costs associated with each project are projected. Explanatory notes 
are provided. Pricing of an Integrated operational system for the SCRR,.ETE, 
and TPF models Is covered In the next subsection. 

Cost Estimate - Integrated DOTS System. Based on the pricing exercises under 
Categories 1 and 2, pages lV-1 and IV-7t a forward pricing analysis was per- 
formed in Phase II, projecting the costs of a horizontal extension of the 
SCRR, ETE, and TPF models to form an Integrated CNET modeling system. Fig- 
ures IV-9 and 10. pages iV-17 and iv-18» formed the basis for the estimated 
number of users as well as the personnel and supporting facilities required. 
Cost savings and avoidance resulting from Implementation are covered In the 
Estimated Cost Savings and Avol dances » page lV-19. 

Due to the number of variables and lack of operational experience, this 
pricing exercise must be considered highly speculative. DOTS Phase III will 
provide experience In an operational environment, enabling a more precise 
definition. However, the projected costs presented here should be suffi- 
ciently accurate to support tactical planning. 

The results of the cost analysis are presented In Figure IV-7, page IV-11. 

The following comments and definitions will assist In Interpretation. 

a. The hardware-software system supporting the DOTS system Is 
projected as taking three possible forms categorized as 
Alternatives 1, 2, and 3. 

Alternative 1 assumes one large system supporting all users. 
It ffl^y or may not support other applications. Users woul<l 
Interface with the system through mall or phone requests. 
Response would normally be through the mailing of a prater 
run. 

Alternative 2 assumes each user will depend upon a system 
located In close geographic proximity. As with Alternative 1, 
Interface would be through phone and mall service and, In 
addition, courier. Alternative 2 assumes these multiple 
systems are in existence and the DOTS application would be 
in addition to existing ones. Alternatives 1 and 2 assume 
utilization of a Navy or Government data system. 
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Alternative 3 assumes a11 users w111 Interface with a coimon 
time-sharing service. In addition to phone and mail, their 
Interface would Include significant direct Interaction through 
a teleprocessing display terminal. Alternative 3 cost pro- 
jections assume a commercial time-sharing service. 

\>* To develop the data base for the Phase II and III validation 
tasks, a specific test site was used. The Data Base Develop- 
ment line, Figure IV-7, estimates the costs necessary to 
develop the data required for additional users. 

c. User and systems personnel costs were based on the manpower 
billet costs for life cycle planning contained In the publica- 
tion. Navy Military Manpower Billet Cost Data for Life Cycle 
Planning Purposes, Bureau of Naval Personnel, Personnel Re- 
search Division, Personnel Systems Research Branch, April 1972, 
NAVPERS 15163. 

d. FY 76 Is assumed as the year of Implementation and FY 77 as 
first year of full operation. Costs were arbitrarily adjusted 
for Inflation since 1972 and Into future years. 

DEVELOPMENT OF DATA SOURCES. Figure IV-8 Illustrates the purpose of this sub 
section In the block entitled Lack of Data or Technique. 
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One of the Phase II, Task 5, objectives was to identify those key data or 
technique voids resulting in rejection of any of the twenty-one Phase I model- 
ing candidates. The two subsections to follow address both the data base and 
technique support for the accepted models, as well as for those rejected. The 
absence of a data void does not necessarily imply that a data base could 
easily be developed for a given model, but that one could be reasonably con- 
structed. 

Data Base and Techniques - Accepted Models . There were no significant data 
base or technique voids identified during development of the SCRR, ETE, or 
TPF models. Originally, the Phase II validation runs were to have been ac- 
complished with contrived data parameters, with use of live or historical 
data taking place during Phase Ill's predictive validation. However, due to 
the availability of historical data and the cooperation extended by COMTRALANT 
(Norfolk), it was possible to complete Phase II's logic validation using actual 
data. 

To be effective, the ETE model does not require a constructed data base. 
Normally, Its data requirements will be developed to fit the unique Indi- 
vidualized Learning System (ILS) problem being addressed. In any event, there 
Is a paucity of historical data pertinent to ILS programs. This lack is not 
a problem, but as supporting ILS data are derived through use of the ETE model, 
it Is reconmended that they become a part of the data base. 

The SCRR and TPF models both require basic data elements concerning training 
resources such as instructors, classrooms, student loads, course descrip- 
tions, etc. These elements form the majority of the data base developed during 
Phase II. Discussions of these data elements are included in Sections tl and 
IV of Volume II, and Section V of Volume III, of this DOTS Phase II Final 
Report. 

In developing the parameters driving the TPF model, a significant quantity of 
statistical data was developed, analyzed, and documented. These statistical 
runs will have value to many groups other than those specifically interested 
in the TPF model. They are documented in Volume III. 

Data requirements and procedures for the three accepted models are covered in 
depth in DOTS Phase II Final Report, Volumes II and III. There were two major 
issues that evolved from the Phase II development effort, one a recommendation 
and the other a concern. 

These were as follows: 

a. RecormieniJation - After completion of DOTS Phase III, and 
after gaining some operational experience with the models 
and data base, serious consideration should be given to in- 
corporation of the data base into the Navy Integrated Train- 
ing and Resources Administration System (NITRAS), currently 
being developed as an application under the Naval Training 
Information System (NAVTIS). 

This transfer will facilitate future distribution of the models 
across the CNET functions and is compatible with the intent of 
NITRAS development. The delay in this transfer is desirable to 
pennit a period of stabilization in the structure of the DOTS 
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data base, as well as for the current development projects 
impacting NITRAS. 

b. Concern - The TPF model development required analysis of raw 
data pertinent to individual student performance. Although 
the end objective was a collective indicator of performance 
by multiple students with common performance or background 
characteristics, the starting point had to necessarily be 
Individual student records. Future use of the TPF model, 
and enhancement to its validity through reassessment of its 
key parameters, are both dependent on this type of analysis. 

The second and third level modeling efforts projected in 
Figures IV-5 and IV-6, pages IV-8 and IV-10, will require 
similar data reductions. 

It is anticipated that a growing concern will ultimately re- 
sult in various privacy regulations that could have a detri- 
mental Impact on the Navy's authority to collect and store 
the tvpes of data required to support future efforts exempli- 
fied by the TPF project's statistical analyses. 

Data Base and Techniques - Rejected Models . Of the seventeen candidate models 
rejected for development in Phase II, only one was eliminated due to a data or 
technique void. This was the Physics of Learning model which was concerned 
with the relationship between training media and the learning rate of students. 
The original twenty-one candidate models were primarily concerned with the 
fundamental concern of basic resource management, and the essential elements 
of their quantification are defined in Volumes II and III of this report. 

Significant voids were identified during development of reconmendatlons for 
future multi-level modeling efforts. The void resulting in rejection of the 
Physics of Learning model Is only one example of a family that must be filled 
If the ultimate objective of an integrated multi-level modeling system, based 
on more complex data elements than basic resources. Is to be achieved. This 
family of data elements is concerned more with learning effectiveness factors 
than with training resources. No void Is perceived to exist in mathematical 
modeling techniques if the effectiveness parameters can be quantified. 

The DOTS Phase I and II studies highl;|ghted the difficulty of a deterministic 
Identification of learning factor type data voids, and the techniques required 
to fill them. However, if the Phase I recommendations pertinent to training 
measurement and control are to be applied through the development of higher 
level decision models, these difficulties must be overcome. As a result of 
this identified need, a. task has been Incorporated into the DOTS Phase IIA 
project, and is entitled Educational Technology Assessment Model (ETAM). One 
of the subtasks under ETAM is to provide a definitive assessment of the 
family of data elements categorized under the heading of Learning Conditions, 
and to Incorporate them into a model design amenable to computerization. 

The ETAM effort will result in specific recommendations and plans for quanti- 
fication of the following data voids to a level enabling their Incorporation 
as mathematical model parameters: 
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a. Training requirements expressed 1n terms of skills required 
on the job after completion of training, 

b. Student and/or graduate competencies resulting from applica- 
tion of various existing training technologies, as well as 
those defined through futuristic speculative exercises. 

c. Learning paradigms reflecting empirically derived assumptions 
pertinent to learning theory. 

d. Cost formulae establishing a basis for forward estimation of 
the true costs of a given or projected training strategy. 

, ETAM will also address the techniques required to accomplish the preceding. 

Sumnary - Development of Data Sourc es. No major data or technique voids, 
pertinent to the current DOTS modeling effort, were Identified during the 
Phase II effort. Detailed descriptions of the data base developed during 
Phase n are Incorporated as appropriate In Volumes II and III of this 
Phase II report. 

In projecting long-range Implications, significant data voids were defined. 
The ETAM task of the DOTS Phase IIA project Is designed to provide a resc? 
tlon to these voids. This r'»«;olut1on, In turn, will facilitate development 
of models Incorporating p^rauicters based on various human factors, as well 
as training resources. 

PROJECTED OPERATIONAL RESPONSIBILITY 

The purpose of this subsection Is to provide a suggested pattern of 
responsibility and user assignments for the SCRR, ETE, and TPF models, and 
their supporting data base. These assignments are critical to the success of 
the DOTS modeling Implementation. The types of decisions the DOTS models are 
designed to support are, for the most part, non-programmable and require the 
subjective judgment of a human being for their final resolution. 

The recommended patterns of assignments are divided Into two major categories. 
The first Is concerned with the organizational assignment of the models to 
'arlous CNET functions; the second with operational and managerial responsi- 
bilities within a given function, as appropriate to a decision analysis system. 
Although not all Inclusive, the following key considerations are essential to 
any practical Imp 1 omental ion of the assignment reconmendatlons. 

a. Currently, there Is no standardized formatting of all data 
elements pertinent to training resources across CNET functions. 
Prior to a horizontal extension of models, this standardization 
must be completed. 

b. Parameters supporting the models were based on various statis- 
tical analyses of data pertinent to the Norfolk FLETRACEN. 
Some of these parameters are directly transferable to other 
centers or functions, but In some cases new analyses will be 
required. Sufficient information Is provided In this report 
for generation of these analyses. 
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c. In developing recoiimendatlons* 1t was assumed that the decision 
command levels identified currently have, or will be granted, 
the authority to make the training resource decisions supported 
by the models. 

CATEGORY 1 ORGANIZATIONAL ASSIGNMENT. Category 1 provides a proposed distribu- 
tion of organizational units utilizing model outputs. The organizations iden- 
tified are those of a size and/or complexity standing to gain a significant 
benefit from application of the DOTS models. Omission of a given function is 
not intended to imply that use of the DOTS models would have no value, but that 
implementation cost may not be justifiable. 

Figure IV-9, page IV-17, presents a proposed control and user structure. For 
the most part, the figure is self-explanatory. However, the levels of control 
require explanation. The following levels are indicated on the chart: 

Level 1 CNET 

Establishes the guiding principles and objectives of the total DOTS 
decision analysis system. 

Level 2 CNET STAFF FUNCTION 

Inspects to ensure that the models are being utilized effectively 
across the using commands, the principles defined by CNET are being 
supported, and the CNET objectives are being achieved. 

Level 3 CNET STAFF FUNCTION 

Converts CNET principles and objectives to operational systems tasks 
and assigns these tasks to CNTECHTRA, Memphis. CNTECKTRA, ADS, will 
have primary responsibility for horizontal extension of the software 
and hardware supporting the DOTS integrated system. 

Level 4 CNET FUNCTIONAL COMMANDS 

Level 4 is Identical to Level 2, except for the level at which 
Inspection and control is taking place. 

The control boundaries on the right of the chart represent those pertinent to 
the technical control of the system, as opposed to management use of the 
models. Such areas as data protection, software modification, security codes, 
etc., are Implied by this type of control. 

CATEGORY 2 ASSIGNMENTS WITHIN FUNCTIONS. Category 1 was concerned with users 
within the CNET organization and control of the integrated system. This sub- 
section is concerned with the organization of responsibilities within a using 
function's structure. It is this structure, in combination with the Category 
2 relationships, which will make the Integrated system supportive of a good 
decision analysis process. 

Figure IV-IO, page IV-18, identifies the key personnel required and pro- 
vides descriptive data pertinent to their activities and responsibilities. 
An estimate of the man months required on an annual basis is also included. 
These estimates formed the basis for the estimate of user costs in Figure 
IV-7, page IV-ll. 
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The sj^stems support personnel Identified at the bottom of figure IV-10, do not 
represent user resources, but increased requirements of the ADS activity. 

ESTIMATED COST SAVINGS AND AVOIDANCES 

4-1 A?®S^?^"°" Implement the DOTS models at the eighteen user locations 
Identified in the preceding Projected Operational Responsibility subsection, 
should be based on a thorough cost versus savings and avoidance justification. 

justification and decision should not be made prior to the completion of 
DOTS Phase III and an assessment of actual operating results at the Norfolk 
Fleet Training Center test site. 

The Phase II DOTS validation exercise did provide sufficient indicators for a 
preliminary projection of cost savings and avoidances. Necessarily, this 
analysis had to be based primarily.on subjective opinion and assumptions, due 
to lack of objective operational data, 

POTENTIAL FOR CAVINGS. The most significant potential for savings through 
integration of the DOTS models into the CNET decision analysis process is in 
the following areas: 

a. Increased efficit.v-y in the utilization of available training 
resources. 

• Increased capacity. 

• Reduced resource requirements. 

b. Optimization of student flow through the training system. 

• Improved quota control - reduced "no-shows". 
§ Improved application of substitute quotas. 

• Reduced Incompletes and "set backs". 

• Reduced "wait time" prior to course convenings. 

c. Permit realization of efficiencies projected for the Indivi- 
dualized Learning System strategy. 

t Reduced student flow restriction due to resource contention. 

• Optimized ILS resource requirements. 

There are other significant tangible and,.. intangible benefits anticipated from 
integration of DOTS models, but those listed are the ones that can be reason- 
ably quantified for justification purposes at this time. 

TRAINING COST STANDARD MODEL. As a first step In projecting savings, a cost 
model of the recruit and specialized training activities was developed. 
Figure IV-11, page IV-20, portrays this model In terms of student flow and 
resource requirements. Although based on actual historical and planning data, 
the standard is not intended to reflect any given fiscal year. The model 
does not contain all CNET students and resources, but only those providing 
potential savings through application of the DOTS models. 

Figure IV-11 is self-explanatory. Factors 1n this cost standard will be re- 
flected in the savings analysis in Figure IV-12, page IV-22. 



ERIC 



Bisi m Mmi 



TAE6 REPORT NO. 12-2 



1^ Hf CWUtT ^ P 



TV1»S TRAIMINO 




[ ' 11,000 | # t ATTWITIOW H 

^ 11.0% ^ mArriiiTMN h 



1 iO.r C^A^.ML'H.'^M 



1^ twiTiAttniu n '"' JiicatywooR6ssiowr 



i TOTALS^ 



131 



WOAVS 



U7.00t 



+ 



MM 



1 L 



MANVtAMOyfniOtWTlOAD 

i 



0^^*000#000 
011^000,000 



jai.iM^ 
"iiiiSr 



I M10 I ♦ I 1 ♦ [ 



009,000 

'"'SiiiT"*' 













7.m 



607^ 



I 



n,i4e 







SrUDiM 

mcnuiT 


Tcom 

MCtALItfO 




1 vmM 















■AH VEAIV TflAINtMOirArP LOAD 




UOO 

""ijoo** 




MUTAAVIlAliVIAm 






DO 




TOTAL 

1 





low 

"TJoT 



omooaooo 





TOTAL CQfTi 






RCOIUtT 


MOALltfO 















MIO^O^OO^J 
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SAVINGS AND AVOIDANCES. A five year projection of savings and avoidances 
expected to result from the DOTS implementation, was developed based on the 
cost standard model in Figure IV-ll, and the Phase II model validation out- 
come. The resultant was used as a financial base for assessing probable 
actual Impact over the sane period. 

Estimates of net savings ranged from $1,975,768 for the first year of imple- 
mentation, to $17,969,727 for the fifth year. Figure IV-12 presents, in 
addition to the financial base, a matrix covering the first year of implemen- 
tation and extending through the fourth year of fully integrated operations. 
In addition to time, the matrix anticipates three possible levels of indivi- 
dualized Instruction curriculum, as well as the three DOTS operational alter- 
natives outlined In Figures IV-7 and IV-9, pages IV-ll and IV-17. 

The following comments should be considered in interpreting Figure IV-12: 

a. Savings and avoidances were predicted for specialized training 
direct and student costs only. Recruit training was excluded, 
since the models will have their greatest impact on specialized 
training. 

b. The fact that only a portion of training costs can be reasonal 
controlled was considered. 



c. The cost of impleme ting and operating the three DOTS models 
was subtracted to arrive at a net savings ©mount. These DOTS 
costs Included manpower as well as computer systems. 

d. The estimates assume implementation of the three DOTS models 

by eighteen users as defined in Figures IV-9 and 10, pages IV-17 
and IV- 18. 

e. The financial base amounts were significantl-y- discounted to 
anticipate unpredictable events tending to reduce potential 
savings and avoidances. For example, those projected for the 
first year, the year of phased implementation, were reduced 
by 90 percent. Each of the five years was assigned a "most 
probable achievement" factor expressed as a percentage of 
the base amounts. 



1st year - 10% 
2nd year - 20% 
3rd year - 35% 
4th year - 55% 
5th year - 65% 



The learning curve in using the models and the effect of delayed 
Impacts were also considered in developing these percentages. 

f. As previously stated, this exercise should be repeated after 
more objective data have been developed during Phase III pre- 
dictive validation of the three models. 
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ASSUMPTION 

SAWINSS AND AVOIDANCE PROJECTED FOR SPECIALIZED STDDtNT AND 
CONTROLLABLE DIRECT COSTS ONLY 



DISCRIPTION 



OPTIMIZE TRAINING RESOURCr UTILIZATION 

IK^Tftucffm §tAFF.OPttMI« mt — 

CLASS SCHfDULING - OPTIMI^F STUDENT FLOW 

• iMPROvrn oiiOTA ifiNiwii (RtiHirrn no-shows anh imcwivid 

APPLICATION OF M)H-.rill)Tf nilOIA,) 

• RLDUCF INCOMPLETFS AND S£I -BACKS (CAUSE 1 DENT I U CATION) 

• OPTIMIZE I.L.^,. "IIUOINI PROGRESSION 
INSTRIICTIONAl ntVICIS AND FACllITIFS - OPTIMIZE USE 



O PTIMIZE MANPOWER UTILIZATION 

UfeoUtE PRE-TRA1H1N6 UAlT TIME 
IMPROVED MATCH OF HAN TO TRAINING 
IMPROVED MATCH OF TRAINING CAPABILITY TO FLEET NEED 
NOTE: NO SAVINGS INCLUDED UNDER THIS CATEGORY, AirnOUaH 
SIGNIFICANT AMOUNTS ARE ANTICIPATED. 



PROJECTED 


COST SAVINGS AND AVOIDANCES - POTENTIAL FROM SPECIALIZED TRAINING 


CATEGORY 


MODEL 


APPLICATII 


3N CODES 


% OF TOTAL SPECIALIZED CURRICULUM IN I.L.S. MODE 


NO CLAIM 


CLAIMED 




50% 


7S% 


SCRR 

ETE 

TPF 


*5 + 8 

/ii5 + 6 + 8 
^5+7+8 


#1 + 4 
#1 +2+3 


7,517,812 
7,691,118 
4.559,923 


5,011 ,875 
15,382,237 
3.307.601 


2,505,937 
23,073,355 
2,067,543 


TOTAL BASE POTENTIAL 


19,768.853 


23,701 ,7^3 


27,646,835 



i PROBABLE NET COST SAVING AND AVOIDANCE - FIRST FIVE YEARS - DOTS EXPENSE CONSIDERED 



ALTERNATIVE - 1 
ALTERNATIVE - 2 
ALTERNATIVE - 3 


1,976,064 
1,975,768 
1,976,037 


2,369,350 
2,369, 0b4 
2,369,323 


2,763,862 
2,763,566 
2,763,835 






SECOND YEAR - FULLY OPERATin(^Af_ 




20% OF BASE LINE MI NL 


S DOTS COST 


ALTERNATIVE - 1 
ALTERNATIVE - 2 
ALTERNATIVE - 3 


3,953,055 
3,952,464 
3.952,989 


4,739,627 
4,739,036 
4,739,561 


b,bi;b,6bl 
5,528,060 
5,528,585 






THTnn vrao - FULLY OPERATIONAL 




35% OF BASE LINE MINUS DOTS COST 


ALTERNATIVE - 1 
ALTERNATIVE - 2 
ALTERNATIVE'- 3 


6,918,383 
6,917,792 
6,918,317 


8,294,884 
8,294,293 
8,294,818 


9,675,677 
9,675,086 
9,675,611 






FOURTH YEAR - FULLY OPERATIONAL 




55% OF BASE LINE MINUS bOTS COST 


ALTERNATIVE - 1 
ALTERNATIVE - 2 
ALTERNATIVE - 3 


10,872,154 
10,871,563 
10,872,088 


13,035,227 
13,034,636 
13,035,161 


15,205,044 
15,204,453 
15,204,978 




ALTERNATIVE - 1 
ALTERNATIVE - 2 
ALTERNATIVE - 3 



36,568,695 
36,566,035 
36,568,404 



43,844,486 
43,841,826 
43,844,195 



51,142,961 
51,140,301 
51,142,670 



FIGURE IV-12 PROJECTED COST SAVINGS AND AVOIDANCES 
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_EiETH .YEAR - FULLV OPERATTONAI 




65% OF BAsfe l\nt Minus Wts cost 


ALTERNATIVE - t 
ALTERNATIVE - 2 
ALTERNATIVE - 3 


12.849.039 
12,848,448 
12,848,973 


15,405,398 
15,404,807 
15,405,332 


17,969,727 
17,969,136 
17,969,661 
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DEVELOPMENT SCHEDULE 

A suggested schedule for implementation of /nulti-level DOTS modeling 
system was presented in Figure IV-6, page IV-10. The major decision and 
phase schedules are covered here. 

a. Complete Phase III 

SCRR, ETE, and TPF Validation/Verification 1 Oct 75* 

b. Implement DOTS Phase III Models 

Eighteen User Locations From 1 Oct 75 - 1 Oct 76 

c. Study and Design Level II Feasibility (EJAM) 

From 1 Sep 74 - 1 Aug 75* 

d. Develop and Validate Level 11 Model (ETAM) 

From 1 Aug 75 - 1 Aug 77* 

e. Implement Level II Model (ETAM) From 1 Aug 77 - 1 Feb 78 

f. Justify Level III Potential From 1 Feb 76 - 1 Oct 76* 

g. Design, Develop, and Validate Level III Model 

From 1 Oct 76 - 1 Oct 79* 

h. Complete DOTS Multi-Level Integrated System 

All defined implementation activities 1 May 80 

The suggested schedule is based on a phased implementation strategy. 
Asterisks (*) indicate major decision points. Each phase should be validated 
and cost justified before a decision to move to the next. 

IDENTIFICATION OF AREAS OF EDUCATION AND TRAINING REQUIRING ADDITIONAL RESEARCH 

The DOTS Phase I functional analysis of the naval training system resulted 
In Identification of a number of problem areas requiring additional research if 
they are to be resolved. These were documented in the Phase I report^S and sub- 
sequently amplified m a summary document '° produced by the Training Analysis 
and Evaluation Group located at the Naval Training Equipment Center, Orlando, 
Florida. These recommendations will not be repeated in this section. 

Two major gaps identified in Phase I were highlighted again during Phase II. 
These were related to predicting student failure and training effectiveness. 
Both will be covered in this subsection. 

STUDENT STATISTICS. Phase II's TPF model was designed to project training 
program yield in numbers of students graduating, Significant numbers of 



T5 

Design of Training Systems, Phase I Final Report , TAEG Report No. 12-1, 
December 1973« 

16 Design of Training Systems, Phase I Sufrniarv Report . TAEG Report 11-1, 
December 1973. " 
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student background were conducted during Phase II to establish 
tiin ratlsl^' °^ and their training compll 

Based on the Phase II experience, it is suggested that additional research in 
correlating various background experiences or characteristics to actual per- 

mfr? ^5® ^^t?^* "^^sult in a significant enhance- 

ment of the Navy s ability to achieve a good match of men to training and to 
subsequent assignments. ^ 

IfSn'^nl^ilLJf?^®^!^ ^'^^^ '^^^ "lo^^l °" an experimental basis, and observa- 

bling°usld ?S de\oX^Vf'^ background element is 

oeing used to determine training assignments, tend to validate this suggestion. 

the^"?s?^"fnrffio'??l Projections for implementing the DOTS models across 
tn efmn;?I luril ^^5 jscessary resources to perform the analysis required 
to support the TPF model at the new user locations. -"^'J''*^^ requirea 

TRAINING EFFECTIVENESS. The three Phase II models do not include oredictivp 

??I?n^Il!^?i}? ^^^^^ ^° « feasibility determination and pre- 

dlf nfMoi°?f%^!'IS3 J?** e^^^ectiveness model, as well as a more precise 
;f the additional research required to develop effectiveness oara- 

ES^SJ^^T^' '"^i;'''""'" ^^^^^ °^ validation. This effort 1^ en tit led' 
Educational Technology Assessment Model, ETAM. entitiea. 
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SECTION V 



CONCLUSIONS AND RECOMMENDATIONS 

SECTION INTRODUCTION 

-ffnJ^® 5111^5^! ?t ^^S^^*^" ^ ^5 ^° sumnarize the results of the DOTS Phase 
?h?ttee'J?l2^:ri? ?Jls''?:?o??"^""'"^ ^"^ recon^endatlons are coverSd'fn 

CONCLUSIONS 

s1ons^2"e d?ai!l! '^""''^"'^ ^ '"^ resultants, the following conclu- 

I^! Naya] Education and Training System can Improve the effective- 
fhlLnl .if ^?^^s1ons pertinent to resource planning and control 
!;r««S?^"SV^f?'"P"*^^^"^ mathematical models. This can be 
syS lnd%actic«f^^" ^^"^^^ management 

IJ^fc?5;nf ^^o?""^ !!:^ logically valid and do perform 

as designed. Pnase III predictive validation will prove their 
degree of accuracy in reflecting actual events. 

c. Sufficient historical and operational data are available in 

models ^ ^^^^^^^ *° operational implementation of the DOTS 

RECOMMENDATIONS 

a. Assuming successful predictive validation in Phase III, initiate 
model's controlled Implementation of the SCRR, ETE, and TPF 

b. Concurrent with model implementation, make the changes to the 
management system required to permit effective implementation 
of resource decjs ons. This implies granting various levels 

2L;rSi-!"fu°^^lf^*^^ "'^r® a^^^orlty over the use of training 
resources than they now have. 

c. Continue the current thrust towards a multi-level modeling 

!^If:®? organized decision analysis process 

spanning the CNET Command. 
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